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ABSTRACT 

In order to examine empirically the priori ties that 
different groups place on a common set of evaLuative characterist ics 
as they apply to a specific problem, a survey of five principal 
groups composed of professional evaluators, E.S.E.A. Title III staff, 
graduate students, local school staff, and administrators was 
conducted. An eighty-five item instrument employing a seven-point 
scale for applicability ratings was developed. Analysis suggests 
differences between some groups on some of the items. The aevice and 
methodology reported should improve communication between evaluators 
and clients, help focus evaluations in an efficient manner, and 
identity specific decisions that characterize the evaluative 
enterprise. A survey problem statement, instructions for completing 
the survey, distribution of responses for each item, and other 
relevant statistics are included. (Author/LR) 
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Planning for the evaluation of educational institutions, programs, 
and products requires critical decisions to be made on the allocation of 
available resources within a diversity of priorities that reflect the 
evaluator, the client, and possible audiences. Efforts to examine this 
aspect of educational evaluation have not been reported in the literature. 
Specifically* attention has not focused on the characteristics that are 
Judged most applicable by different audiences when a specific evaluation 
problem is viewed across a common set of evaluative characteristics. 
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Itespite the state of the ait, and the so-called crude level of 
development that describes many of its instruments, an obvious need and, in 
so many cases, a mandate for evaluation exists within the educational 
community today. When technologies are developed to satisfy that need* 
communication has to occur among several groups, especially during the 
planning and reporting of the evaluation. Language is always a problem, 
but when the educational evaluator, drawing as he must from many disciplines, 



1 Paper presented at the American Educational Research Association Annual 
Convention, New York City, 1971. 
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attempts to plan and communicate with other groups, he frequently employs 
a terminology that operates with little efficiency for communication, 
Consequently, planning is difficult and communication harder. 

The failure of some evaluators to recognize the limitations of their 
efforts and to communicate their constraints to their clients has resulted 
in an evaluative oversell that has frustrated and embittered many, 

This study sought to develop a defensible categorization of 
evaluative characteristics and to use them as the basis for a survey to 
describe the priorities that six groups within the educational community 
held on a particular program evaluation problem. 

A. Method 

1 , Subjects 

The six groups surveyed were: (1) Undergraduate teacher3-in- 

training, (2) Teachers, (3) Graduate students, (4) School and project 
administrators, (5) Evaluators, and (6) Community members. It was originally 
intended by the investigators to differentiate between school administrators, 
board members, and project administrators, but insufficient returns 
required vhe categories to be reduced to a single one, school and project 
adninistrators. 

One hundred eleven respondents (Ss) returned usable survey 
instruments. Seventeen of the Ss were teachers-in-training; fifteen were 
teachers; eleven were graduate students; twenty-one were school or project 
administrators ; twenty-one were evaluators; and twenty six were community 
members . 
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Instrument 



To insure a survey instrument that possessed an acceptable level 
of content validity, a logical analysis of available evaluation models and 
theories was conducted and a list of eighteen characteristic elements 
composed. These elements described those aspects of evaluative procedures 
that were judged by the investigators to appear most frequently in the 
literature on evaluation. The eir^teen characteristics were: (1) Decision, 

(2) Relationships, (3) Judgments, (4) Standards, (5) Description, (6) Cost, 

(7) Personnel, (8) Design, (9) Rationale, (10) Instrumentation, (11) Analysis, 
(12) Conclusions, (13) Language, (14) Information, (15) Services, (16) Coor- 
dination, (17) Planning, and (18) Objectives, 

Positive statements describing the components of each of the 
characteristics were then composed. A total of eighty-five statements were 
written. In most cases, the number of statements composed for each 
characteristic varied according to the apparent complexity of the character- 
istic, The logical loading of the eighty-five items on the eighteen 
characteristics is given in Table 1, 



Insert Table 1 About Here 



All Ss were presented with a brief written description of a 
program evaluation problem situation (See Appendix), and were asked to 
judge the applicability of each of the eighty-five statements to that 
evaluation problem. When making this judgment, Ss were told to respond 
from the point of view they felt would most probably be theirs if they 
attended an evaluation planning meeting such as the one depicted in the 
problem statement. Judpaents of applicability were made on a six -point 
scale that was defined as Hot Applicable, Low, Somewhat, Moderate, 
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Considerable, and High Applicable, Respondents were instructed to omit 
it 2 ms if they felt they had no reasonable basis for rating them or if they 
felt they did not understand them* The problem statement and the set of 
eighty-five items together with proportions of responses over the entire 
sample appear in the Appendix* 

3, Data Reduction 



Responses were collected on machine scorable answer sheets and 
were rendered for computer analysis* 

4. Analysi s 

Data were analyzed for each of the six groups, Tne proportion 
of responses in each category for each item over all groups were computed* 
Keans and standard deviations were also computed for the eighteen character- 
istics (subscores) for each of the six groups* 



5, R esults 



A comparison of the group means on the eighteen characteristic 
subscores showed no apparent disparity* The difference between the group 
means on the characteristic subscores ranged from a low of ,24 on the 
Decision subscore to 1,04 on the Objectives subscore* 



Despite the similarity in group means on the sixteen character- 
istics, when the subscores were ordered by meani for each of the six 
groups and the rankings compared, some patterns appeared* Allowing that 
the differences between rankings for some groups were very small, only the 
three highest and the three lowest rrnkings are reported. Table 2 shows 
which characteristics were ranked highest and lov>e8t by each of the six 
groups , 
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Insert Table 2 About Here 



Of interest is the consistent appearance of the Rationale and 
Conclusions characteristics (9 and 1$) as two of the three highest ranked 
characteristics in five of the six groups represented. Rationale's appear- 
ance in all six groups 1 top rankings may be the result of the use of only 
three, items (8, 13, 30) to represent it and a corresponding response set 
on the part of Ss to recognize t^a obvious applicability of focus and 
raison d T etre in an evaluative effort or any other, for that matter. In a 
similar fashion, the emphasis on Conclusions may be both a reflection of an 
implicit '‘utility" criterion that works as a critical yardstick in questions 
of evaluative applicability as well as an indication that, whether they be 
drawn by the evaluator or the decision maker, the groups sampled saw 
Conclusions as an aspect of the evaluative effort that is highly applicable 
in program evaluation. 

The Cost characteristic, attaining a high ranking by community 
members and a low one by school and project administrators, deserves 
mention. One reasonable interpretation of the reversal would be that the 
community members were more concerned with their pocketbooks than the 
administrators were, or alternatively, that the idmiristrators questioned 
in this study had relegated fiscal concerns to <;he level of moderately 
inapplicable as far as planning for an evaluation was concerned. The latter 
interpretation would seem to defy usual expectations. 

In order to determine whether any considerable disparity existed 
between groups, those items on which at least thirty per cent of the dis- 
tribution fell above and below the midpoint were selected. On this basis, 
sixteen of the eighty-five itero3 qualified. An examination waf then made 



5 



to determine whether the item also appeared to differentiate between at 
least two of the six groups. Of the sixteen items, five seemed to present 
considerable between group differences. 

Items such as Number 7, 'The evaluator will present the committee 
with a range of instruments which might be used to collect data," and 
Number 57, 'The evaluator will take precautions to insure that the findings 
generalize across time," showed satisfactory overall variability according 
to the thirty per cent decision rule, but they did not differentiate 
between any of the six groups. On the other hand, items 22, 37, 41, 49, 
and 61 elicited both high overall variability and considerable between group 
disparity. The group means and standard deviations for these items are 
summarized in Table 3. Since each of these items may be characterized by 
high variability in response in each group, exact response proportions are 
not presented. The extremely high score or low score is not of primary 
interest in each case, rather, the appearance of clusters of means is 
suggested. 



Insert Table 3 About Here 



6. Discussion 



The results of the survey point to several problems with the 
instrument and with the description of priorities in the planning of program 
evaluations. It appears that unconstrained ratings of items may contribute 
to a positive response bias on the part of Ss. The problem definition 
placed no explicit constraints on the respondents* Time, money, or 
availability of personnel, for example, were not constrained in any way. 

It appears that in thJc hypothetical ,r best of all possible worlds", the 
individuals studied in this survey did not venture far from saying that all 
eighteen characteristics were at least moderately applicable to the problem. 
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It may be that the presence of no discrepancy between the groups on 
the eighteen characteristics, as well as on a considerable majority of the 
items, suggests that the use of an assumed role designation as a point of 
view from which applicability is to be judged does not provide a powerful 
basis for discriminating between the applicability of evaluation character- 
istics* 



Moreover, the respondents may have been constrained by the 
investigators 1 own biases in item selection and the creation of an instrument 
of considerable length* Respondents required between twenty and thirty 
minutes to complete the form. 

These inconclusive results seem to imply that titles or role desig- 
nations that identify individuals participating in a program evaluation 
planning session may be an unreliable basis for predicting evaluative 
preferences. An individual^ decision on the relative applicability of 
particular characteristics of an evaluative study may not be as closely 
related to his role designation as they are to other personal and ideosyn- 
cratic factors. Replication of these results would lend credence to 
this notion. 

One clear pattern , of response that seems to appear in these results 
is thac on nine of the sixteen characteristics computed* the respondents 
who classified themselves a teacher s-in* -training (Group 1) scored the 
characteristic as more applicable than did any of the other groups surveyed. 
Although the differences do not possess aty statistical reality, the 
strong pattern of response may suggest that teacher s-in-training who have 
had only academic experience in the planning of evaluations, may exhibit 
a tendency to give high applicability to the theoretical dimensions of a 
program evaluation. 
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On the other h^nd, on eleven of the sixteen characteristics, consnu- 
nity members (Group 6) judged the characteristic less applicable than did 
any of the other groups. In no instance did the community group give the 
highest rating to a characteristic, as compared to other groups, (See 
Table 4,) 



Insert Table 4 About Here 



Interpretation of this pattern of response may point to the heavy 
role that jargon played in the wording of the items on the instrument. It 
appears that when faced with unfamiliar jargon, the respondent may adopt 
either a standard role* response (judging everything in the same direction), 
or he may consistently avoid the extremes of judgment and choose some 
scale midpoint. It would appear that the community members surveyed in 
this study chose the former mode and consistently rated items as having 
middle to low applicability. 

It seems clear that both the individual items and characteristics 
contained in this form possess an applicability that will vary as the 
evaluation problem and its constraints do. Informal respondent interviews 
have suggested that one of the utilities of this form may lie in its 
reduction to a checklist for use by those engaged in the planning of 
evaluations. With the removal of items that appear primarily as indicators 
of internal consistency (21, 28, S3, 43, 48, 52, 57, 58, 63, 64, 70, 75, 76), 
the remaining statements should function as a valid checklist that would be 
especially useful for those untutored in the tradeoffs implied by most 
evaluative planning decisions. Employed then as an aid to conmunication be- 
tween evaluator and client, early, explicit decisions and understandings of 
the weights that will be allocated to such matters as judgment, causality, 
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and instrumentation, for example, may facilitate both communication and 
problem focus while helping all parties avoid the frustrations of un- 
fulfilled expectancies which have frequently been the trademark of 
program evaluations. 
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APPENDIX 



Evaluation Characteristics Survey 
Problem Stateme nt 

Wessex is a comprehensive 9-12 high school located in a suburban 
community in the Northcentral United States. Serving essentially a well- 
settled, middle and upper middle class community, the school's present 
°tudent population numbers approximately 2,300. A 3 the result of a request 
put forth by a local community action group and supported by both the 
teachers f and administrators* local professional associations, the school 
board and district staff allocated funds to be used to support an evaluation 
of the school's social studies program by an outside team of evaluation 
specialists. 

As one important part of the planning of the evaluation, a steering 
committee composed of key district staff, administrators, teachers, students, 
community members, and evalua :ors was formed and requested to compose a 
list of points that should be observed during the planning, conducting, and 
reporting of the evaluation, A preliminary list of those points has been 
prepared and distributed to th* committee for their judgments on the appli- 
cability of each of the items. 

Please assume that you hold a seat on the committee described above, 
and from your own point of view, complete the attached form according to 
/struct ions. 
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INSTRUCTIONS FOR COMPLETING THE EVALUATION CHARACTERISTICS SURVEY 



1 . Please Note : 

A. Use ONLY PENCIL . 

B. Do not write on the question sheets, 

C. Record all responses on the answer sheet, 

2. In column number 1 of the space labeled S TUDENT NUMBER blacken the 
number that most clearly identifies the role you would most likely 
assume if you were to attend the evaluation planning meeting just 
described. Choose only one . 



ROLE COKE 

Teacher-In-Training 0 
Graduate Student 1 
Teacher 2 
School Administrator S 
School Board Member 4 
Project Administrator 5 
Evaluator 6 
Community Member 



3. Enter the following information in the space labeled STUDENT NUMBER 
under the designated column. 



COLUMN # 


HIGHEST LcGREE HELD 


core 




Elementary Diploma 


0 




Secondary Diploma 


1 


o 


Bachelors Degree 


2 


L 


Masters Degree 


3 




Advanced Certificate 


4 




Doctorate 


S 


COLUMN tf 


Have you ever participated in 


0 = No 


3 


an evaluation planning session? 


1 = Yes 


Complete the 


information box on SEX. 
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6. Please read each of the 8S statements describing various characteristics 
of an evaluation and in the corresponding space on your answer sheet 
rate the applicability of that characteristic to the evaluation 
problem Just described. In rating the applicability of each character- 
istic, use the following scale: 

| 1 l 2 | 3 1 4 L § | 6 1 

NOT LOW SOMEWHAT MODERATE CONSIDERABLE HIGHLY 

APPLICABLE APPLICABLE 

If you feel yon do not understand an item or if you feel you have no 
reasonable basis for rating it, leave the item blank. 

When you have completed all items, please return all materials. 

Thank you. 
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ITUS USED IN THE SURVEY 
AND THE PROPORTIONS IN THE 
TOTAL RESPONSE DISTRIBUTION (N=lll) 

1 . In cases where instruments must be developed, sufficient time will be 
allocated for development, try-out, and revision, before the instru- 
ments are used as part of the evaluation. 

NA .02 L ,00 S ,05 H .08 C .29 H A .54 OMIT .03 

2. Certain phases of the implementation of the evaluation will be 
scheduled by the teachers and adninistrators in the school. 

N A .04 L .05 S .12 M .19 C ,fl HA .27 OMIT .03 

3. The recommei .ons given by the evaluators will be both global 
and specific. 

N A .03 L ,01 S .08 M .15 C .31 H A .41 OMIT .02 

4. The evaluation report will have a section devoted exclusively to 
recommendation;.* which the evaluators will make concerning program 
improvement . 

NA .01 L .03 S .05 M ,14 C .23 H A ,54 OMIT .01 

5. The evaluation plan will Include alternate days for data collection 
and alternative methods of collection to allow for unforseen problems. 

N A .05 L .06 S .13 M .22 C .24 H A .27 OMIT .01 

6. Various reporting nodes will be used in presenting the final Infor- 
mation package. 

N A ,03 L .09 S .14 M .14 C .36 HA .24 OMIT .00 

7. The evaluavor will present the committee with a range of instruments 
which might be used to collect data. 

N A .04 L .09 S .17 H .32 C .24 HA .14 OMIT .00 

8. The specific questions to be considered during the course of evaluation 
will be clearly stated. 

NA .02 L .00 S .03 M .14 C ,17 HA .64 
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OMIT .00 



9. Conclusions drawn by the evaluators will be clearly supported by the 
analysis of the data. 

N A .00 L .03 S .02 M .14 C .23 HA .59 OMIT .00 

10. The origins of the standards used in making judgments will be stated. 

NA .05 L .02 S .06 M .09 C .32 HA .46 OMIT .00 

11. The evaluation will collect credible information that has utility 
for decision makers. 

NA .02 L .02 S .04 M .09 C .31 HA .52 OMIT .01 

12. Various methods of data collection will be used for each type of data. 

NA .02 L ,15 S .16 H .22 C .20 H A .24 OMIT .01 

13. The rationale of the evaluation will be clearly stated in both the 
planning stage and in the final report. 

NA .00 L .02 S .04 M .14 C .31 HA .50 OMIT .01 

14. The normal operation of the school will not be unduly disrupted by 
evaluation activities. 

N A .01 L .03 S .05 M ,15 C .32 HA .43 OMIT .00 

15. The language of the evaluation reports will be conversational and 
free of jargon. 

NA .05 L .05 S .11 H .21 C .29 HA .30 OMIT .00 

16. The description of the program will be organized according to a set 
of appropriate theoretical constructs. 

N A .05 L .09 S ,16 M .24 C .23 HA .13 OMIT .10 

17. There will be a single final report presented to the administration 
and the community. 

NA .07 L .14 S .05 M ,21 C .22 HA_j30_ OMIT .02 

18. Planning time will be provided for the evaluation, 

NA .00 L .01 S .03 M .06 C .27 HA .60 OHIT .03 

19. The analysis procedures will provide an efficient and economical mode 
for reporting trends in the data gathered. 

NA .03 L .05 S .09 H .20 C .37 HA .22 
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20. Judgments will be based upon information whose reliability and 
validity are known. 

N A .00 L .09 S .12 M .16 C .26 H A .36 OMIT .01 

21. Information will be presented which is based upon data collection 
devices which vary in mode and time of collection. 

NA .04 L .06 S .15 M .22 C .32 HA .19 OMIT .03 

22. The evaluation will include statements which imply causality among 
parts of the program. 

NA .08 L .14 S .11 M .23 C .21 HA .17 OMIT .0? 

23. The instruments used in the evaluation will be published standardized 
tests. 

N A .20 L .41 S .18 M .11 C .09 HA .01 OMIT .01 

24. The design of the evaluation will guard against results which may be 
due to factors other than the program itself. 

NA _;A)7_ L .09 S .11 M .15 C .25 HA .30 OMIT .03 

25. The analysis of the data will not be limited to strict statistical 
tests . 

NA .05 L .04 S .08 M .14 C .25 HA .41 OMIT .03 

26. Time and money will be budgeted for consultant services. 

NA .03 L .07 S .13 M .18 C .30 HA .29 OHIT .01 

27. Facilities and personnel of the school will be available for 
implementing the evaluation. 

NA .01 L .03 S .00 M .19 C .35 HA ,41 OMIT .Ot 

28. The evaluation team will use supportive community services whenever 
necessary. 

NA .04 L .05 S .12 H .25 C .23 HA .30 OMIT .01 

29. Inferences made upon the data will be clearly distinguishable from 
reconroendations which the evaluators make. 

NA .01 L .04 S .08 M .15 C .30 HA.j37_ OHIT .05 

30. The data collected by the various instruments will be relevant to the 
questions under consideration in the evaluation. 

NA .02 L .01 S .01 H .06 C .52 HA .58 OHIT .01 
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31. The evaluation will contain a report of judgments made by several 
groups concerning the worth of the program. 

NA .05 L .06 S .11 M .19 C .30 HA .29 OMIT .01 

32. The planning of the evaluation will be a cooperative venture carried 
on by the evaluation team and the committee in consort. 

NA .04 L .03 S .09 M .10 C .32 HA .43 OMIT .02 

33. The evaluator will be required to state the basis upon which his 
judgments of program worth are based. 

NA .02 L .02 S .04 M .14 C .25 HA .53 OMIT .00 

34. Interim evaluation reports will be presented to the committee and 
administration . 

NA .04 L .08 S .10 M .34 C .22 H A .22 OHIT .01 

35. The evaluation will provide information for those in decision making 
positions. 

NA .01 L .01 S .04 M .04 C .31 H A .59 OMIT .01 

36. The evaluator will be responsible for synthesizing the data and drawing 
conclusions relevant to questions asked about the program. 

N A .02 L .03 S .05 M .14 C .41 HA .34 OMIT .01 

37. All statements of program objectives will Indicate a specific behavior , 
a desired level of performance of that behavior, and the conditions 
that will surround that behavior. 

NA .05 L .12 S .13 H .24 C .23 HA .22 OMIT .03 

38. The evaluation team will be the sole author of the final evaluation 
report . 

NA .09 L .14 S .14 M .17 C .19 HA .26 OMIT .01 

39. The evaluation team will communicate well with each other and with 
the connittee. 

NA .01 L .02 S .02 M .12 C .23 HA .60 OMIT .00 

40. A member of the committee will serve as a coordination agent, acting 
as llason between the school staff and the evaluation team. 

NA .06 L .06 S .12 H .18 C .31 HA .27 OHIT .00 
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41. The names of the people who will be making the decisions about the 
future of the program will be made known during the planning phase. 

NA .22 L .16 S .10 M .15 C .18 HA .18 OMIT .01 

42. Part of the evaluation will be presented as an oral report to an open 
meeting of parents, teachers, and administrators. 

HA .03 L .11 S .10 M .20 C .23 H A .32 OMIT .01 

43. The implementation of the evaluation will be coordinated with the 
activities of the school. 

N A .02 L .03 S .06 M .12 C .40 HA .36 OMIT .02 

44. Attempts will be made to control the quality of information collected. 

N A .05 L .05 S .06 M .11 C .30 HA .40 OMIT .03 

45. Recommendations for program improvement will be made informally by the 
evaluators during the final committee meeting. 

NA .11 L .20 S .14 M .20 C .23 HA .13 OMIT .01 

46. The total co3t of the evaluation will be stated in an itemized budget. 

NA .04 L .05 S .07 K .17 C .30 HA .37 OMIT .01 

47. The evaluation will describe the priorities that have been assigned 
to the program objectives. 

NA_jOO_ L .00 8 .08 M .20 C .33 HA .34 OMIT .00 

48. The evaluation will try to find out what aspects of the program are 

causally related. 

NA .07 L .06 S .08 M .20 C .35 H A .21 OHIT .03 

49. The school staff will specify the measurable bjectives that represent 
the intended outcomes of the program. 

NA .06 L .08 S .15 M .25 C .26 HA .16 OMIT .03 

50. The recommendations which the evaluators make will be clearly traceable 
to contingencies found in the data gathered. 

NA .03 L .04 S .06 H .14 C .29 HA .41 OMIT .04 

51. The evaluation team will consist of individuals who 'possess vard»us 
skills, l.e., meansrement specialists, subject matter specialists, etc., 
to insure that the skills and abilities necessary to complete the 
evaluation are available. 

NA .01 L_ 1 02__ S .00 M .09 C .27 HA .59 OMIT .02 
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52* The evaluator will collect judgments concerning the worth of the program* 
NA *05 L >03 S *10 M *25 C *32 H A *22 OMIT .03 

53* The total program will be described in the evaluation so that an out- 
sider could understand "what it was all. about" just by reading the 
report . 

NA *03 L *05 S *09 M .21 C ,31 HA *32 OMIT .01 

54* The total cost of the evaluation will be decided upon through joint 
agreement of the committee and the evaluators* 

NA .07 L ,02 S *09 M *19 C *32 HA .27 OMIT *05 

55. An appropriate design will guide the planning and implementation of 
the evaluation. 

NA .03 L *01 S ,06 M *16 C .40 HA .33 OMIT 

56. The report will contain statements which assign confidence levels to 
research findings given as part of the evaluation. 

NA .03 L ,08 S .14 M .26 C .25 HA .15 OMIT .09 

57* The evaluation will take precautions to insure that the findings 
generalize across time. 

NA *08 L .13 S *14 H .19 C .22 HA *18 OMIT *07 

58. The evaluation will contain a description of the .logical relationships 
among the parte of the program. 

NA .01 L *0S S *08 M *22 C .32 H A ,29 OMIT .03 

59. When the evaluator makes conclusions, he will be responsible for stating 
explicitly the basis upon which he made ^hose conclusions. 

NA ,00 L .05 S .05 M .12 C .28 HA .50 OMIT .01 



60, The final report will contain graphs, charts, and other schema to 
illustrate the quantitative data collected. 

NA .01 L .07 S .10 M .23 C .23 KA .32 OMIT .04 



61 . Copies of the evaluation report will be considered to be the property 
of the adninietration of the school, and they will decide who receives 
the information. 



S *08 H ,14 C .13 
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NA .29 L .18 



HA .15 



0HIT .03 



62. Data collection will occur when data should be collected rather than 
when it is convenient to collect data. 

NA .05 L .11 S .14 M .12 C .22 H A .35 OMIT .03 

63. The committee will share the authorship of all reports concerning the 
evaluation study. 

NA .14 L .20 S .12 M .22 0 .23 HA .06 OMIT .04 

64. The evaluation will report instances where there is discrepancy between 
what was intended to occur and what actually occurred in the program. 

NA .00 L .01 S .05 M .13 C .37 HA .43 OMIT .01 

65. The evaluation will eventuate in predictions about possible future out- 
comes or contingencies. 

NA .02 L .07 S .11 M .23 C .33 HA .20 OMIT .05 

66. The specific decisions that have to be made will be indicated as soon 
as possible. 

NA .03 l i .04 S .09 M .21 0 .28 HA .32 OMIT .05 

67. The evaluation team will maintain a non-threatening and supportive 
atmosphere . 

NA .00 L .02 S .05 M .09 C .26 HA .58 OMIT .01 

68. The evaluation will report instances where what was intended by the 
staff actually occurred. 

N A .04 L .03 S .09 M .14 C .28 HA .42 OMIT .01 

69. There will be a number of school staff assigned as full-time members 
of the evaluation team. 

NA .13 L .17 S .13 M .20 C .23 HA .12 QHIT .04 

70. The evaluation will not intrude upon the normal schedule of the program. 

NA .05 L .08 S .12 H .22 C ,26 HA .26 QHIT .02 

71. The evaluation will develop criterion measures for each objective of 
the program. 

NA .04 L .05 S .11 H .15 C .34 HA .27 OMIT .05 

72. The evaluation team will assume the major responsibility for developing 
measurable objectives that represent the program. 

NA .07 L .15 S .11 M .23 C .24 HA .15 OMIT .04 
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73. Certair. phases of 'the implementation of the evaluation will be carried 
out at the sole discretion of the evaluator. 

NA .13 L .17 S .12 M .26 C .20 HA .08 OMIT .05 

74. Decisions on the allocation of time and money for the evaluation will 
be arrived at through the Joint consideration of the comnittee and 
the evaluators. 

NA .05 L .05 S .12 M .19 C ,33 HA .25 OMIT .01 

75. The evaluator will indicate his criteria for Judgments he makes about 
discrepancies between what was intended and what actually occurred. 

N A .01 L .02 S .05 M .15 C .33 HA .41 OMIT .03 

76. The planned allocation of time for the evaluation will be stated. 

N A .03 L .05 S .10 M .22 C .32 HA ,27 OMIT .01 

77. Some findings of this evaluation will be applicable to other programs 
in the same school setting. 

NA .13 b .07 S .18 M .23 C .23 HA .12 OMIT .04 

78. The evaluation will develop an empirical description of the relation- 
ships among the parts of the program. 

NA .02 L .06 S .12 H .27 C .27 HA ,14 OMIT .12 

79. The evaluation team will use supportive school services whenever 
necessary rather than outside agencies. 

NA .02 L .08 S .14 M .25 C .28 HA .18 OMIT ,0S 

80. Standards used by the evaluator will be acceptable to the local 
decision maker. 

NA .08 L .06 S .13 M .23 C .23 HA .23 OMIT .03 

81. All intended outcomes of the program will be stated in measurable terms. 

NA .09 L .08 S .10 H .17 C .33 HA .21 OMIT .02 

82. A survey of educational research relevant to the program will be 
Included as part of the evaluation report. 

NA .04 L .12 S .20 M .16 C .25 H A .22 OMIT .02 

83. The recommendations presented by the evaluation team will be feasible, 
given the school climate and staff. 

NA .02 L .01 S .07 M .14 C .34 HA .39 OMIT .04 
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84. Tha evaluator will be required to make judgments about the worth of 
the program. 

m .04 L .05 S .12 M .16 C .30 HA .32 OMIT .02 

85. When other groups make judgments about the program, their criteria 
will be stated. 

NA .02 L .02 S .05 M .17 C .29 HA .41 OMIT .05 
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TABLE 1 



Logical Loading of E ight y-Five Items 
on E ighteen Evaluative Character i stics 



Evaluative Characteristics 


Logical Item Loadings 


1. Decision 


35, 41, 66 


2, Relationships 


22, 48, 50, 58, 64, 68, 78 


3, Judgments 


20, 31, 52, 84 


4, Standards 


10, 33, 75, 80, 85 


5, Inscription 


16, 53 


6 , Cos* 


46, 54 


7. Personnel 


39, 51, 67, 69 


3. Design 


24, 55, 57, 77 


9* Rationale 


8, 13, 30 


10 , Instrumencat ion 


1, 7. 12, 21, 23, 44 


11. Analysis 


19, 2 ■ , 56, 65 


12, Conclusions 


3, 4, 9, 29, 36, 59, 83 


13, Language 


6, IS, 38, 63 


14, Information 


11, 17, 34, 42, 45, 60, 61, 82 


1 . Services 


26, 27, 28, 79 


16, Coordination 


2, 5, 14, 40, 43, 62, 70, 73, 74 


17, Planning 


18, 32, 76 


18. Objectives 


37, 47, 49, 71, 72, 81 
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TABLE 2 



Three Characteristics Rated Highest and Lowest 
in Applicability 
by Six Groups 



Group Number and Name Three Characteristics Ranked: 



Highest Lowest 



1, Undergraduate Teachers-in- 
Training 


9* Rationale 
12. Conclusions 
4. Standards 


14, Information 
10, Instrumentation 
13. Language 


2, Teachers 


9. Rationale 
12, Concisions 
17. Planning 


8. Ensign 
14. Information 
13, Language 


3 , Graduate Students 


9. Rationale 
7. Personnel 
12, Conclusions 


10. Instrumentation 
13. Language 
5, Description 


4. School and Project 
Administrators 


9. Rationale 
17. Planning 
7, Personnel 


14, Information 
10, Instrumentation 
6. Cost 


5, Evaluators 


9, Rationale 
17. Planning 
12. Conclusions 


14. Information 
8. Design 
18. Objectives 


6. Comnunity Members 


9. Ratiorale 
12, Conclusions 
6, Cost 


10. Instrumentation 
1. Decision 
13. Language 
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TABU) 8 



G roup M eans and Standard Deviations for 
Items Showing Greatest Variability 



Item 22, The evaluation will include 
among parts of the program. 

Group Number X S.D. 


statements which inply causality 
Group Number X S.D. 




1 


4.59 


1.58 


4 


4.40 


1.50 




2 


3.80 


1.74 


5 


3.61 


1.54 




3 


3.45 


1.44 


6 


3.55 


1.47 


Item 37 , 


AIL statements of program objectives will indicate a 


specific 




behavior. 


a desired level of performance of that behavior, and 




the conditions that will uurround that behavior. 




Group Number 


X 


S.D. 


Group Number 


X 


S.D. 




1 


5.29 


.92 


4 


4.48 


.98 




2 


4,23 


1.54 


5 


3.38 


1,53 




3 


3,91 


1.64 


6 


3,88 


1.51 


Item 41. 


Tl > name 8 


of the people who will be making the 


decisions about 




the future of the 


program will be made known during the 




planning phase. 










Group Number 


X 


S.D. 


Grow Number 


X 


S.D. 




1 


3.88 


1.82 


4 


3.10 


1.84 




2 


3,40 


1.84 


5 


4.48 


1.57 




3 


3,00 


1.84 


6 


2.92 


1.81 



Item 49, The school staff will specify the measurable objectives that 
represent the intended outcomes of the program. 

Group Number X S.D. Group Number X S.D, 



1.02 

1.23 

1.60 



4 

5 

6 



4.33 1.32 
3.67 1,49 
4.12 1.64 
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1 

2 

3 



4.63 

3.67 

4,10 
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TABLE 3 - Continued 



Item 61* Copies of the evaluation report will be considered to be the 
property of the administration of the school, and they will 
decide who receives the information* 



Group Number 


X 


S.D. 


Group Number 


X 


S.D. 


1 


2.47 


1*55 


4 


3.86 


1.90 


2 


3,29 


1*77 


5 


3.42 


2*04 


3 


3.91 


1.70 


6 


2*23 


1.6' 



Group Key: 

1 = Undergraduate Teachers-In-Training 

2 - Teachers 

3 s Graduate Students 

4 - School ard Project Administrators 
3 * Evaluators 

6 = Consnunity Members 
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TABLE 4 



High and Low Means for Groups 
on Sixteen E valuative Characteristi cs 



Characteristic 


High 


Low 


Group Number 


X 


Group Number 


X 


It Decision 


1 


4.94 


6 


4.03 


2. Relationships 


1 


5.05 


5 


4.44 


3, Judgments 


1 


4.90 


6 


4.34 


4, Standards 


1 


5.23 


6 


4.61 


5, Description 


1 


4.76 


3 


3.86 


6* Cost 


3 


5.10 


4 


4.05 


7, Personnel 


3 


5.33 


6 


4.73 


8« Design 


3 


4.55 


5 


4.06 


9, Rational 


1 


5.59 


5 


5.08 


10. Instrumentation 


2 


4.33 


6 


4.06 


11. Analysis 


1 


4.83 


6 


4.37 


12, Conclusions 


1 


5.36 


6 


4.92 


13. Language 


** 


4.33 


6 


3.74 


14. Information 


3 


4.58 


6 


4.14 


15. Service a 


4 


4.92 


5 


4.36 


16, Coordination 


3 


4.66 


6 


4.25 


17, Planning 


3 


5.23 


6 


4.75 


18, Objectives 


1 


4.87 


5 


3.83 



Group Key: 

1 = Undergraduate Teachers-In-Training 
l = Teachers 

3 = Graduate Students 

4 = School and Project Administrators 

5 = Evaluators 

6 = Community Members 
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